The Computation of Normalizers in Permutation Groups

The general problem of computing the normalizer, in a finite permutat ion group G, of a subgroup H has long been recognized as being unusually difficult to solve efficiently. The corresponding problem for centralizers is much easier, although this too has some bad cases. The solution to the centralizer problem is relatively simple and probably difficult to improve upon, whereas there seems to be almost unlimited scope for possible improvements to the normalizer problem. The aim of this paper is to describe some of these improvements which have been successfully implemented by the author. The ability to compute normalizers is important, not only in itself, but because it is potentially an ingredient in other algorithms, such as computing Sylow subgroups of groups or automorphisms of groups. One attempt at a solution of the normalizer problem has been described in Butler (1983). The author 's program uses the same general method, but tries much harder to keep the cpu time as low as possible. More specifically, the general idea is to impose the structure of a tree on the elements of G, and to perform a backtrack search through the tree, looking for elements of G which normalize H. At a given node in the tree, it is often possible to use group-theoretical arguments to show that none of the elements of G lying below that node can possibly normalize H, in which case we do not need to search that part of the tree; in other words, we can chop off the branch at that node, and save ourselves a lot of time. Naturally, the higher the node, the more we chop off, and the more time we save. The improvements introduced by the author have been more of these group-theoretical tests designed to prune the search tree. Of course, the tests themselves introduce certain overheads in terms of both time and space, but the experimental evidence suggests that the time saved overall is enormous in many cases, whereas space is unlikely to be a serious problem. For example, the case in which Butler's algorithm performs worst is when H is a regular group (that is, it acts transitively, with all of its non-trivial elements acting fixed-pointfreely), and G is the whole symmetric group. In fact it becomes impractical in most examples for degrees greater than about 20. The author's algorithm, on the other hand, can cope reasonably quickly with this case for degrees over 100 in many examples. As